Developer demo of alternate wiki store implementation #7329

Jermolene · 2023-03-04T21:08:39Z

This PR introduces two alternate implementations of the tiddler store object:

A smallest possible plain JS wiki store implementation, sharing as much implementation as possible with the existing core implementation with as little code duplication as possible
An implementation based on the WebAssembly build of sqlite3
- Wasm usage from a file URI without requiring external files
- Sql implementations of main wiki store methods
- Pass test suite with sqlite3 store
- Tag indexer optimisations
- Explore further optimisations

Testing with the current prerelease wiki, the sqlite3 implementation is currently significantly slower than the standard plain JavaScript wiki store implementation in boot.js.

Plain JS store:
- mainRender: 89.20ms
- styleRefresh: 4.00ms
- mainRefresh: 12.00ms
sqlite3 store:
- mainRender: 300.00ms
- styleRefresh: 380.30ms
- mainRefresh: 175.80ms

@linonetwo

This is the very barebones beginnings of a demo implementation of an alternate tiddler store. It is not functional. If using the Vercel builds, open developer tools in the browser to see it failing due to the absence of basic wiki methods. The plan is to build it up into the smallest possible plain JS wiki store implementation, sharing as much implementation as possible with the existing core implementation with as little code duplication as possible. It could then serve as the basis for future experiments with wiki stores based on SQLite (@linonetwo), or a custom append only database (@yaisog).

vercel · 2023-03-04T21:08:42Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated (UTC)
tiddlywiki5	✅ Ready (Inspect)	Visit Preview	Dec 11, 2023 8:43pm

btheado · 2023-03-05T00:23:17Z

boot/boot.js

@@ -1466,10 +1466,10 @@ $tw.Wiki = function(options) {
 // Dummy methods that will be filled in after boot
 $tw.Wiki.prototype.clearCache =
 $tw.Wiki.prototype.clearGlobalCache =
-$tw.Wiki.prototype.enqueueTiddlerEvent = function() {};
+$tw.Wiki.prototype.enqueueTiddlerEvent = $tw.Wiki.prototype.enqueueTiddlerEvent || function() {};


Is this change and all the following in this file so these methods can be optionally be defined in the code of the alternate store implementation?

The comment says they "will be filled in after boot". Won't that overwrite the method if it is implemented in the alternate store?

Also with the above it looks like clearCache and clearGlobalCache will also have the same function as enqueueTiddlerEvent if it has an alternate store implementation. Is that intended?

The comment says they "will be filled in after boot". Won't that overwrite the method if it is implemented in the alternate store?

These three methods all have an implementation loaded from wiki.js, but need to be invoked before wiki.js is fully loaded, so we provide this dummy implementation.

Is this change and all the following in this file so these methods can be optionally be defined in the code of the alternate store implementation?

That's the idea. At the moment the additional methods in wiki.js are loaded unconditionally, and so can't be overridden by alternate stores, which is OK for the moment but might need to be revisited

Also with the above it looks like clearCache and clearGlobalCache will also have the same function as enqueueTiddlerEvent if it has an alternate store implementation. Is that intended?

These methods are currently overwritten by the implementations in wiki.js, and as above there's currently no support for alternate stores to override those methods.

The wiki store implementation in boot.js has always used a closure for key state information like the hashmap of tiddlers and shadow tiddlers, which makes clear which wiki methods actually have and need access to the raw tiddler store, and to encapsulate the implementation details of the store. That means that it should be possible to just override those methods that have access to the private data in the closure, and seamlessly take advantage of the implementation of the higher level methods in wiki.js.

Sounds reasonable and I might still be missing part of your point.

By my thinking these 3 lines should have either remained unchanged since overwriting them would only last until wiki.js is loaded and therefore isn't very useful:

$tw.Wiki.prototype.clearCache = $tw.Wiki.prototype.clearGlobalCache = $tw.Wiki.prototype.enqueueTiddlerEvent = function() {};

or each should have their own separate overwrite:

$tw.Wiki.prototype.clearCache = $tw.Wiki.prototype.clearCache || function(); $tw.Wiki.prototype.clearGlobalCache = $tw.Wiki.prototype.clearGlobalCache || function(); $tw.Wiki.prototype.enqueueTiddlerEvent = $tw.Wiki.prototype.enqueueTiddlerEvent || function();

Hi @btheado I think you're right that there's no point in providing an underride for these methods because they will be overwritten later.

pmario · 2023-03-05T08:24:52Z

I always thought, that the TW concept could have "additional" $tw.wiki stores if needed. ...

With the implementation like this there can only be 1 type of store. Is this intended or is it just an experiment?

linonetwo · 2023-03-05T08:49:39Z

In plugin, we always call $tw.wiki, so a singleton is better.

Just need some way to hook into existing $tw.wiki

Jermolene · 2023-03-05T21:57:37Z

I always thought, that the TW concept could have "additional" $tw.wiki stores if needed. ...

I think that's an orthogonal goal, which would be accomplished by reclassifying the existing $tw.wiki instance as the root admin wiki, and allow for a dynamic set of child wikis that can be created and deleted without a restart.

With the implementation like this there can only be 1 type of store. Is this intended or is it just an experiment?

An alternate store along these lines could still expose its own constructor so that further instances could be created.

joshuafontany · 2023-05-07T23:38:51Z

Thanks for starting to sketch this out. I think this would allow tighter integration between TW and data-stores like the Yjs CRDT document(s). :)

See https://github.com/Jermolene/TiddlyWiki5/pull/7329/files#r1126709318

linonetwo · 2023-06-08T12:52:38Z

About how to use sqlite and if it should be a plugin #7531

@joshuafontany

And make the demo storage areas switchable @joshuafontany's implementation was in #7521

linonetwo · 2023-06-22T09:31:20Z

How to support multiple plugins?

I want to create a sqlite plugin that works on nodejs server side, loading tiddler and indexer from sqlite instead of fs, then do swr from fs.
I also want to create a plugin that work on client side, to get indexer&tiddlers using ipc, so we can skip most of parsing on client booting.
And there might be a plugin you want to create, work on client side, using wasm sqlite to cache content...

Jermolene · 2023-06-22T10:24:04Z

Hi @linonetwo the "demo-alternate-store" plugin in this PR is intended to be a starting point for people wanting to write their own custom alternate store plugins. The current implementation only works in the browser but will be extended to work under Node.js as well.

To support multiple alternate store plugins being loaded at the same time we will also need a mechanism for selecting between them.

…pendencies We get a reference to sqlite3 but we're not yet doing anything with it Also note that this approach leads to duplication - there will be two copies of sqlite3.js and sqlite3.wasm in each generated HTML file. The plan is to dynamically retrieve those tiddlers from the store area rather than baking them into the raw markup area

empty.html with the plugin is now 4.1MB

Previously, we were using the existing addIndexer method to piggyback adding our own internal indexers.

At the moment the optimiser returns a list of chainable functions, it would be simpler to just return a single function

pmario · 2023-11-08T11:33:41Z

core/modules/filters.js

 						operation.suffixes.push([]);
 						$tw.utils.each(subsuffix.split(","),function(entry) {
 							entry = $tw.utils.trim(entry);
 							if(entry) {
 								operation.suffixes[operation.suffixes.length -1].push(entry);
 							}
 						});
-					 });
+						});


I think the indent-level here is wrong

Thanks @pmario fixed in 4897248

@pmario

Thanks @pmario

linonetwo · 2024-01-27T10:37:56Z

plugins/tiddlywiki/sqlite3store/init-sqlite3.js

+	};
+$tw.utils.evalSandboxed(sqlite3js,context,"$:/plugins/tiddlywiki/sqlite3store/sqlite3.js",true);
+// Create a Blob URL for the wasm data
+var sqlite3wasm = thisPluginTiddlers["$:/plugins/tiddlywiki/sqlite3store/sqlite3.wasm"].text;


Why this, instead of $tw.wiki.getTiddlerText ?

Context is #7947

linonetwo · 2024-01-27T10:48:31Z

plugins/tiddlywiki/sqlite3store/init-sqlite3.js

+$tw.utils.evalSandboxed(sqlite3js,context,"$:/plugins/tiddlywiki/sqlite3store/sqlite3.js",true);
+// Create a Blob URL for the wasm data
+var sqlite3wasm = thisPluginTiddlers["$:/plugins/tiddlywiki/sqlite3store/sqlite3.wasm"].text;
+var decodedData = window.atob(sqlite3wasm),


I get

Uncaught InvalidCharacterError: Failed to execute 'atob' on 'Window': The string to be decoded contains characters outside of the Latin1 range.

when mimicking your usage.

@linonetwo, notice Jeremy's tiddlywiki.info is encoding the file as base64. The atob can't decode it if it wasn't encoded.

{ "file": "sqlite3.wasm", "encoding": "base64", "fields": { "type": "application/wasm", "title": "$:/plugins/tiddlywiki/sqlite3store/sqlite3.wasm" } }

@btheado I have a few tries, even I change encoding to utf8, when I use $tw.wiki.getTiddlerText, I always get the base64 encoded test. Same even set "type": "text/text".

I tried this because 30MB wasm become 80MB of string after become a tiddler and I want to reduce it.

(Update: fixed by #7948)

To be clear, TiddlyWiki doesn't currently support true binary tiddlers; we only support base64 encoded pseudo-binary tiddlers.

linonetwo · 2024-01-27T14:00:55Z

boot/boot.js

@@ -2438,6 +2471,7 @@ $tw.boot.initStartup = function(options) {
 	$tw.utils.registerFileType("image/svg+xml","utf8",".svg",{flags:["image"]});
 	$tw.utils.registerFileType("image/vnd.microsoft.icon","base64",".ico",{flags:["image"]});
 	$tw.utils.registerFileType("image/x-icon","base64",".ico",{flags:["image"]});
+	$tw.utils.registerFileType("application/wasm","base64",".wasm");


This is important. Without this, wasm tiddler will be 2x larger, and can't properly loaded.

vercel bot deployed to Preview March 4, 2023 21:12 View deployment

btheado reviewed Mar 5, 2023

View reviewed changes

Merge branch 'master' into demo-alternate-store

7fdd8a5

vercel bot deployed to Preview May 10, 2023 21:14 View deployment

Merge branch 'master' into demo-alternate-store

6233710

vercel bot deployed to Preview May 10, 2023 21:27 View deployment

Remove unneeded override

b6bc197

See https://github.com/Jermolene/TiddlyWiki5/pull/7329/files#r1126709318

vercel bot deployed to Preview May 10, 2023 21:30 View deployment

saqimtiaz mentioned this pull request Jun 5, 2023

Pr for demo alternate store #7521

Closed

linonetwo mentioned this pull request Jun 8, 2023

[BUG] With 8k+ tiddlers, in a NodeJS wiki everything is Slow #7151

Closed

Merge branch 'master' into demo-alternate-store

3561318

vercel bot deployed to Preview June 22, 2023 08:20 View deployment

Incorporate @joshuafontany's plain JS wiki implementation

1d0b928

And make the demo storage areas switchable @joshuafontany's implementation was in #7521

vercel bot deployed to Preview June 22, 2023 08:39 View deployment

vercel bot deployed to Preview June 22, 2023 13:30 View deployment

Restructure the sqlite3 store as a separate plugin

0546a14

vercel bot deployed to Preview June 22, 2023 14:07 View deployment

Make rawmarkup code dynamically load dependencies

146a22b

empty.html with the plugin is now 4.1MB

vercel bot deployed to Preview June 22, 2023 14:26 View deployment

Test code to exercise the database

449e227

Jermolene added 3 commits October 25, 2023 17:49

Refactor indexer implementation

dc94ed8

Previously, we were using the existing addIndexer method to piggyback adding our own internal indexers.

Cache allTitles and allShadowTitles

3a4f5b8

Fix sqlAllTitles wrongly including shadow tiddlers

e9d640b

vercel bot deployed to Preview October 25, 2023 17:06 View deployment

Jermolene added 2 commits October 27, 2023 10:31

More indexes

01e1882

Enable performance instrumentation

a58f119

vercel bot deployed to Preview October 27, 2023 09:35 View deployment

Jermolene added 2 commits October 27, 2023 14:10

Don't use LEFT JOIN unless we have to

e4af21a

Remove all plugins to simplify benchmarking

d982658

vercel bot deployed to Preview October 27, 2023 13:14 View deployment

SQL Console: Add number of rows returned

8c1f7a6

vercel bot deployed to Preview October 30, 2023 10:06 View deployment

Merge branch 'master' into demo-alternate-store

9493084

vercel bot deployed to Preview November 7, 2023 10:30 View deployment

Jermolene added 2 commits November 7, 2023 10:34

Revert attempt at optimising filter execution

02f3065

At the moment the optimiser returns a list of chainable functions, it would be simpler to just return a single function

Minor refactoring

d7f0c5c

vercel bot deployed to Preview November 7, 2023 20:48 View deployment

Jermolene added 2 commits November 7, 2023 21:08

Refactor filter compilation to allow SQL engine to optimise queries

6997c61

Refactor filter compilation into its own source file

5d20e98

vercel bot deployed to Preview November 8, 2023 09:33 View deployment

pmario reviewed Nov 8, 2023

View reviewed changes

Fix indentation

4897248

Thanks @pmario

vercel bot deployed to Preview November 10, 2023 21:32 View deployment

Merge branch 'master' into demo-alternate-store

6983564

vercel bot deployed to Preview December 11, 2023 20:43 View deployment

Jermolene mentioned this pull request Jan 2, 2024

Introduce "Multi Wiki Server" Plugin #7915

Draft

20 tasks

linonetwo reviewed Jan 27, 2024

View reviewed changes

linonetwo mentioned this pull request Jan 27, 2024

fix: wasm can't be loaded properly #7948

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Developer demo of alternate wiki store implementation #7329

Developer demo of alternate wiki store implementation #7329

Jermolene commented Mar 4, 2023 •

edited

vercel bot commented Mar 4, 2023 •

edited

btheado Mar 5, 2023

Jermolene Mar 5, 2023

btheado Mar 6, 2023

Jermolene Mar 6, 2023

pmario commented Mar 5, 2023

linonetwo commented Mar 5, 2023

Jermolene commented Mar 5, 2023

joshuafontany commented May 7, 2023

linonetwo commented Jun 8, 2023

linonetwo commented Jun 22, 2023

Jermolene commented Jun 22, 2023

pmario Nov 8, 2023

Jermolene Nov 10, 2023

linonetwo Jan 27, 2024

linonetwo Jan 27, 2024

btheado Jan 27, 2024

linonetwo Jan 27, 2024 •

edited

Jermolene Jan 27, 2024

linonetwo Jan 27, 2024

Developer demo of alternate wiki store implementation #7329

Are you sure you want to change the base?

Developer demo of alternate wiki store implementation #7329

Conversation

Jermolene commented Mar 4, 2023 • edited

vercel bot commented Mar 4, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pmario commented Mar 5, 2023

linonetwo commented Mar 5, 2023

Jermolene commented Mar 5, 2023

joshuafontany commented May 7, 2023

linonetwo commented Jun 8, 2023

linonetwo commented Jun 22, 2023

Jermolene commented Jun 22, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

linonetwo Jan 27, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Jermolene commented Mar 4, 2023 •

edited

vercel bot commented Mar 4, 2023 •

edited

linonetwo Jan 27, 2024 •

edited